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abstract 

An auto-instructional program on elementary probability was pre- 
sented in scrambled and properly ordered sequence respectively to two groups, 
each of which numbered 18 freshman psychology students who were classified 
according to their prior mathematical ability. Students were permitted to 
proceed once through the linear program at their own pace, and were given 
a criterion test immediately after completing the learning session. The 
sequence of items had no significant effect on (a) time required for 
learning, (b) error score during learning, (c) criterion test score, and 
(d) time required for criterion test. Prior mathematical ability had a 
significant effect on error score during learning and on criterion test scores. 
There was no significant interaction between the type of item sequencing 
and prior mathematical ability. 
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INTRODUCTION 






Teaching machines have been characterized by three supposedly 
important properties; 

1. The subject matter is broken down into small, carefully ordered 
items, allowing the student to respond explicitly to each item. 

2. The student is informed immediately after his response to each 
item whether his response was correct, and thus has a means 
of immediately correcting a wrong response. 

3. The student proceeds on an individual basis in accordance with 

his own rate of learning. 

In an experiment conducted by Roe with college freshman engineer- 
ing students, it was found that if a carefully ordered sequence of learning 
items was prepared, students learned equally well whether they were required 
to compose their responses, make multiple -choice selections, or give no overt 
responses at all. There was no apparent difference in student learning if 
the sequence was presented by a machine, a programmed textbook, or a lecturer. 
It seemed that the sequence of items alone was important. However, the 
importance of the careful ordering of items became suspect when it was 
discovered that a student, who failed to read the introductory instructions 
of a programmed textbook, read down the page instead of from page to page 
so that the sequence of items he saw were numbered 1, 40, 79, 118, 157, 

2, 41, 80, 119, 158, 3, 42, and so on. This student still managed to get a 
high score on the criterion test. 

This report deals with an experiment designed to determine whether 
the presentation of a proper sequential ordering of related subject items 
does affect the terminal performance of a student differently than does 
the presentation of a random ordering of the same subject items. Stated 
as a hypothesis; The mean performance in a criterion test of students 
who have studied a proper sequential ordering of related subject items 
will be equal to the mean performance on the same criterion test by stu- 
dents who have studied a random ordering of the subject items. The 




hypothesis is based on the use of subject items which are related, with 
one item normally depending on a preceding item, and on the students' 
terminal performance, rather than rheir performance on intermediate items. 
A variable which could affect the hypothesis is the number of items used 
in one learning session. It is obvious that if the program consisted of 
only three randomly ordered items, many students could memorize them, 
mentally unscramble or reorder them, and perform adequately on a terminal 
test. If the program consisted of 200 items, the problem of mentally 
storing and unscrambling them should be much more difficult. In this 
first experiment, the number -of- items variable was not explored. Only 
enough items to make up what would normally be a one -hour learning session 

(71 items, in this case) were used. 

Shortly before this experiment was conducted, the work of Gavurin 
and Donahue^ on logical sequence and random sequence was brought to the 
author' s attention. Gavurin and Donahue used a program of 29 items, 
in a logical sequence for one group of subjects and in a randomized sequence 
for a second group. The latter sequence was divided into three blocks of 
10, 10, and 9 items, with randomization occurring only within each block. 
Subjects in the second group, who presumably were not matched for verbal 
ability or prior knowledge of the material, were required to repeat all 
missed items within a block before proceeding to the next block. A 
criterion test was not administered until one month after the learning 
session. Gavurin and Donahue found that subjects who studied the ran- 
dom sequence made significantly more errors than those who studied the 
logical sequence. However, after one month there was no significant dif- 
ference in the amount of retention of the material between the two groups. 

It was anticipated that the current experiment would shed some light 
on the sequencing variable by (a) scrambling larger blocks of items, (b) 
choosing subject matter which had a minimum of “word** responses 
that may not require logical sequencing, (c) accounting for prior ability 
of subjects, (d) eliminating the repetition of missed items, which may 
tend to exaggerate the error score, and (e) eliminating the leveling effect 
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of time on long-term retention by administering the criterion test immedi- 
atcly after the learning session. 

SUBJECTS 

A group of 36 freshman psychology students were classified into 
upper, middle, and lower thirds according to their prior mathematical 
ability as indicated by quantitative scores on the American Council on 
Education (ACE) college entrance board examinations. Within each third, 
students were randomly assigned to each sequencing method. No pre-test 
was administered because previous studies with the program used indicate 
that lower division college students have little prior knowledge of the subject 
matter, and that the presence of such prior knowledge does not correlate 
with terminal performance after studying the programmed items. 

PROGRAM MATERIAL 

The learning items consisted of 71 frames taken from the program 
on elementary probability developed over the past two years by members of 
the Teaching Systems Research Project at the Department of Engineering, 
University of California, Los Angeles. The concepts covered include 
relationship between information and degree of certainty, deterministic 
vs. probabilistic problems, probability ratio, additive law of proba- 
bility, multiplicative law of probability, sampling with and without 
replacement, and mutually exclusive and independent events. 

Each frame, printed on a 4- by 6-lnch card, consisted of a learning 
item followed by a statement requiring multiple -choice completion. (See 
Figures 1 and 2. ) The student had to remove a patch of opaque rubber cement 
(to which powdered graphite had been added) below one of the possible 
answers, thereby recording his choice as well as uncovering the correct 
response. 

PROCEDURE 

Students were gathered in a classroom and given instruction sheets 
and boxes containing the learning items. Half the students received boxes 
with an ordered sequence of items, and the other half received boxes with 
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THE PROGRAM CONSISTS OF A SEQUENCE OF 71 FRAMES 
(4”x 6" CARDS) ARRANGED IN A BOX. EACH ITEM 
REQUIRES A MULTIPLE -CHOICE COMPLETION. 

FIGURE 1 




THE STUDENT REMOVES THE MASKING MATERIAL BELOW THE 
ANSWER HE CHOOSES, UNCOVERING THE CORRECT ANSWER 

FIGURE 2 



4 



scrambled sequence of items. Immediately on completing the program, 
each student was given a subjective questionnaire and a criterion test. Times 
for completing the learning session and the criterion test were recorded. 

The boxes of cards were subsequently examined to find the number of errors 
made during the learning session. 

STATISTICAL TREATMENT 

Chi-square tests of normality of distributions were made for all 
variables and Bartlert' s test for homogeneity of variances was made for 
all pertinent variances. In no case were the assumptions of normality or 
homogeneity of variances rejected, though it was recognized that for the 
sample sizes involved these tests were not powerful. Also t-tests were 
made to ascertain that the three mathematical aptitude strata were signif- 
icantly different from one another and that, within each stratum, the groups 
were not significantly different from one another. 

Analyses of variance were then performed on the following data: 
time required for learning, error score during learning, time required for 
criterion test, and criterion test scores. The two variables in each of these 
analyses of variance were the method of sequencing items and the students' 
mathematical aptitude. 

Since programmed instruction is designed to bring most students up 
to a high level of performance on criterion items, it was expected that the 
distribution of criterion test scores would be skewed towards the high end 
(negatively skewed). Though this skewing was not large enough to negate the 
normality assumptions (for the limited size of the sample), non-parametric 
tests were also used on the criterion test scores. In all cases, the non-para- 
metric test results were consistent with the analyses of variance results. 
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RESULTS 



The row and column means and standard deviations for the variables 
are listed in Table I. The F ratios, degrees of freedom, whether the F 
ratio is significant at the a~ . 05 level, and the level of a at which the F 
ratio becomes significant (for those who might properly protest against an 
arbitrary choice of a) are listed for the various hypotheses in Table II. 

The sequence of items had no significant effect (at the . 05 level) 
on the dependent variables, nor was there any significant effect on the 
interaction between the sequence of items and prior mathematical aptitude. 

However, prior mathematical aptitude did have a significant effect on the 
error scores and criterion test scores. 

Though not significant, it is noted that the mean criterion test score of 
the students using the scrambled sequence is higher than that of the students 
using the ordered sequence. 

CONCLUSIONS 

Considerable importance had been attached to the careful sequencing 
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of auto-instructional items by most program writers ; and it was thought that 
. the student mentioned in the introduction, who, in an earlier experiment, had 
followed an improper sequence and still scored high on the criterion test, must 
have been unusually gifted. The results of this small-scale experiment, however, 
seem to indicate that college level students may not require the careful sequenc- 
ing of auto- instructional items as had previously been supposed. A clue to 
the possible reason comes from a conversation with one of the students immed- 
iately after the conclusion of the experiment. 



Student: 

Experimenter: 

Student: 



Experimenter: 

Student: 



What kind of program was that? 

Why do you ask? 

I was finding all kinds of things I didn' t know and 
couldn' t answer and was getting curious about what 
was going on. 

What did you do? 

Well, I kept looking for the information for the items 
I couldn' t answer, and when 1 found them later on I 
felt very glad. 
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TABLE 11 



ANALYSES OF VARIANCE RESULTS 





DP 


Significance 


P ratio 


'^i 


^2 


At Of = . 05 


At Of approx. 


Time Required for Learning 












Sequence 


0.30 


1 


30 


NS 


0.65 


Aptitudes 


0.36 


2 


30 


NS 


0. 70 


Interactions 


2.42 


2 


30 


NS 


0. 10 


Error Score During Learning 












Sequence 


0.64 


1 


30 


NS 


0. 55 


Aptitudes 


4.9 


2 


30 


S 


0.02 


Interactions 


1.65 


2 


30 


NS 


0. 20 


Time Required for Criterion Test 










Sequence 


0.04 


1 


30 


NS 


0.95 


Aptitudes 


0.46 


2 


30 


NS 


0. 70 


Interactions 


0.11 


2 


30 


NS 


0.90 


Criterion Test Scores 












Sequence 


K 

2.32 


1 


30 


NS 


0. 15 


Aptitudes 


5. 11 


2 


30 


S 


0.02 


Interactions 


0.64 


2 


30 


NS 


0. 55 



Possibly, a closely ordered sequence of items, in which the informa- 
tion for completing a given item has been supplied in immediately preceding 
items, can be followed easily and without much more than short-time recall 
attention. Presenting items out of sequence possibly introduced a task- 
oriented anxiety which was subsequently relieved in a moment of revelation 
when a missing clue was discovered. That the students using the scrambled 
sequence of items were possibly more alert is indicated by the fact that the 
five comments concerning typographical errors in the program were made by 
students in this group. 

It is difficult to generalize from an experiment using a limited number 
of students studying a narrowly defined subject area in an ad hoc situation. It 
would be interesting to examine the scrambled versus ordered sequence 
question with students of various age levels, with programs of different length 
and in different subject areas, and with retention tests made at spaced intervals 

of time. 

The Teaching Systems Research Project has scheduled an experiment 
for September, 1961, in which approximately 200 college freshman engineer- 
ing students will be exposed to various branching procedures, a linear sequence 
of items, and a scrambled sequence of items. There will be approximately 
200 items in the linear and scrambled sequences. 
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